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Abstract 

Background: We studied a score for assessing basic transthoracic echocardiography (TTE) skills exhibited by 
residents who examined critically ill patients receiving mechanical ventilation. 

Methods: We conducted a prospective study in the 16 residents who worked in our medical-surgical ICU between 
1 May 2008 and 1 November 2009. The residents received theoretical teaching (two hours) then performed 
supervised TTEs during their six-month rotation. Their basic TTE skills in mechanically ventilated patients were 
evaluated after one (Ml), three (M3), and six (M6) months by two experts, who used a scoring system devised 
for the study. After scoring, residents gave their hemodynamic diagnosis and suggested a treatment. 

Results: The 4 residents with previous TTE skills obtained a significantly higher total score than did the 12 novices 
at Ml (18 (16 to 19) versus 13 (10 to 15), respectively, P = 0.03). In the novices, the total score increased significantly 
during training (Ml, 13 (10 to 14); M3, 15 (12 to 16); and M6, 17 (15 to 18); P< 0.001) and correlated significantly 
with the number of supervised TTEs (r = 0.68, P < 0.0001). In the overall population, agreement with experts 
regarding the diagnosis and treatment was associated with a significantly higher total score (17 (16 to 18) versus 
13 (12 to 16), P= 0.002). A total score > 19/20 points had 100% specificity (95% confidence interval, 79 to 100%) 
for full agreement with the experts regarding the diagnosis and treatment. 

Conclusions: Our results validate the scoring system developed for our study of the assessment of basic 
critical-care TTE skills in residents. 

Keywords: Goal-oriented bedside echocardiography; Basic critical-care echocardiography skills; Non-cardiologist 
intensivists; Hemodynamic assessment; Intensive care unit 



Background 

Echocardiography is increasingly used to evaluate the 
hemodynamic status of patients in the ICU or operating 
room. Transthoracic echocardiography (TTE) and trans- 
esophageal echocardiography (TEE) provide a broad range 
of cardiovascular parameters for rapidly understanding the 
causes and mechanisms of circulatory failure [1-8]. In 
many studies, echocardiography was found to contribute 
substantially to the goal-oriented treatment of circulatory 
failure [9-14]. Both TTE and TEE used as tools for 
hemodynamic management benefited therapeutic decisions 
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in patients receiving intensive care [15-21] or undergoing 
surgery [22-25]. Echocardiography is therefore among the 
recommended tools for managing patients with shock [26]. 

The main limitation of echocardiography is that an ex- 
perienced operator is required. Formal teaching pro- 
grams are a crucial prerequisite to the development of 
ultrasound techniques in critical care [27,28]. Experts 
agree that basic critical-care echocardiography skills 
must be taught to all intensivists [29]. Several studies 
have established that non-cardiologist intensivists can 
acquire these basic skills after a short period of training 
[9-11,30-35]. 

Determining whether trainees have acquired the necessary 
echocardiography skills is a major issue. There is no consen- 
sus about the number of supervised echocardiograms a 
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trainee must perform to become sufficiently skilled [29] . A 
standard scoring system with a cutoff value indicating that 
basic skills have been acquired would be helpful [36] . Such 
a scoring system has been validated for TEE in critically 
ill patients [37]. A recent study that used this score in 
ICU intensivists showed that 31 supervised TEEs in mech- 
anically ventilated ICU patients were needed to reliably 
achieve competence in advanced critical-care TEE [38]. 
To our knowledge, these two studies are the only ones to 
use such a standard scoring system. However, both studies 
focused on TEE, although TTE is often the first-line ultra- 
sound method used for hemodynamic assessment [36]. 
Moreover, neither study evaluated whether higher scores 
predicted better treatment decisions by the trainees. Fi- 
nally, the scoring system used in these studies includes 
quantitative measurements, whereas basic critical-care 
echocardiography involves only semi-quantitative mea- 
surements [29]. 

Here, our objective was to validate a score for asses- 
sing basic TTE skills in trainees examining mechanically 
ventilated ICU patients. We also identified the score cut- 
off above which trainees made appropriate decisions re- 
garding hemodynamic management. 

Methods 

Study protocol 

We conducted a prospective observational study in the 
15-bed medical-surgical ICU of a general hospital be- 
tween 1 May 2008 and 1 November 2009. The protocol 
for standard management of circulatory failure was the 
protocol used routinely in our ICU. Standard manage- 
ment involved routine TTE at circulatory failure onset 
and as needed thereafter. In particular, no other devices 
are used in our ICU to assess hemodynamic status of 
the patients. No changes in the standard management of 
circulatory failure occurred in the ICU over the study 
period. Therefore, according to French law on medical 
research, informed consent from the patients or next of 
kin was not required for this observational study. 

Residents 

We included the 16 residents who worked in the ICU dur- 
ing the study period. Their specialties were anesthesiology 
(n = 6), cardiology (n = 2), emergency medicine (n = 4), 
and other (n = 4). They had previously completed 5 ± 2 ro- 
tations of 6 months each. We divided the residents into 
two groups: novices (n=12) had no theoretical and/or 
practical experience with TTE, whereas skilled residents 
(n = 4) were cardiologists or certified echocardiographers. 

Training 

TTE was performed using a Sequoia Siemens C-512 
(Siemens Medical Solutions, Malvern, PA, USA) equipped 
with an Acuson 3V2c probe (Acuson Corp, Mountain 



View, CA, USA). Training was supervised by two board- 
certified echocardiographers with at least two years of 
experience at level-three competence according to the 
American Society of Echocardiography [39]. All residents 
received a two-hour theoretical teaching on TTE, designed 
to show them a selection of typical TTE recordings, dur- 
ing the first week of their six-month rotation in the ICU. 
Then, residents followed the hands-on training session by 
performing TTEs repeatedly during their six-month rota- 
tion in the ICU under the supervision of one of the two 
experts, who provided teaching as appropriate. For this 
purpose, the residents performed all TTEs ordered by the 
attending physician in mechanically ventilated (volume 
assist-controlled mode) patients with acute circulatory 
failure defined as systolic arterial pressure < 90 mmHg 
despite adequate fluid resuscitation [26], admitted during 
their six-month rotation in the ICU. Residents could per- 
form several TTE examinations on the same patient dur- 
ing their hands-on training sessions. As far as possible, 
patients were placed in left lateral decubitus for TTE 
examination. Otherwise, they were placed in the 30° semi- 
recumbent position according to the standard manage- 
ment of the patients in our unit. The use of sedatives and 
neuromuscular blockers was at the discretion of the 
attending physician. We chose to investigate only mechan- 
ically ventilated patients in order to standardize the condi- 
tions under which TTE was performed. 

Evaluation of the residents and scoring system 

After one (Ml), three (M3), and six (M6) months, the 
residents were evaluated during a TTE examination in- 
volving the acquisition of apical, subcostal, and paraster- 
nal views. Patients with poor echogenicity during TTE, 
as assessed by the experts, were not considered for the 
evaluation. The TTE used for the evaluation was per- 
formed in a patient who was not among the resident's 
patients and on whom the residents had never previ- 
ously performed TTE examination during their hands- 
on training sessions. Nevertheless, the resident could 
have access to all patients' medical records at any time. 

At each study time point, the residents were evaluated 
by both experts using the scoring system detailed in 
Table 1. The echocardiographic key-findings the trainees 
should be able to identify consisted of a real time evalu- 
ation. Then, the residents gave their hemodynamic diag- 
nosis and suggested a treatment. Based on their TTE 
examination and on the data in the patient's record, resi- 
dents had to choose among the following hemodynamic 
diagnoses: hypovolemia in the case of large respiratory 
variation in inferior vena cava diameter; impaired left 
ventricular ejection fraction, and/or right ventricular fail- 
ure in the case of marked dilation of the right ventricle. 
They also chose among the following therapeutic op- 
tions, according to the therapeutic algorithm used in our 
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Table 1 Scoring system used to evaluate transthoracic echocardiography skills 
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Total score 


Quality of TTE views 










Left parasternal long axis 


Not recorded 


Not optimal 


Optima 


/2 


Left parasternal short axis 


Not recorded 


Not optimal 


Optima 


/2 


Apica four-chamber view 


Not recorded 


Not optimal 


Optima 
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Apical two-chamber view 


Not recorded 


Not optimal 


Optima 
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Subcostal four-chamber view 


Not recorded 


Not optimal 


Optima 
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IVC view 


Not recorded 


Not optimal 


Optima 


12 
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Semi-quantitative measurements 3 










Right ventricular dilation 
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Agreement for presence but 
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Agreement for presence and degree 
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Pericardial effusion 


Disagreement for presence 


Agreement for presence but 
disagreement for degree 


Agreement for presence and degree 


/2 


Respiratory variation in IVC diameter 


Disagreement for presence 


Agreement for presence but 
disagreement for degree 


Agreement for presence and degree 


/2 


Left ventricular ejection fraction 


Disagreement for presence 


Agreement for presence but 
disagreement for degree 


Agreement for presence and degree 


/2 

/8 
/20 











disagreement or agreement with the assessment of two experts. 
IVC: inferior vena cava; TTE: transthoracic echocardiography. 

ICU: volume expansion in the case of hypovolemia, va- 
sopressors when there was right ventricular failure or 
when the acute circulatory failure could neither be at- 
tributed to hypovolemia nor to left ventricular dysfunc- 
tion, and/or inotropes in the case of left ventricular 
failure. TTE was then repeated by one of the two ex- 
perts, who assessed the quality of the resident's TTE and 
the relevance of the proposed diagnosis and treatment. 

The scoring system was based upon the one developed 
previously for TEE [37] but included neither quantitative 
measurements nor scoring of the diagnosis and treat- 
ment. As shown in Table 1, the first part of the score 
assessed the quality of the TTE views, which was rated 
as follows: 0, view not obtained; 1, view not considered 
optimal by the experts; and 2, view considered optimal 
by the experts. The second part of the score assessed the 
semi-quantitative measurements needed to answer the 
clinical questions relevant to basic critical-care echocar- 
diography, as defined when the study was designed 
[27,40]: right ventricular dilation (none, moderate, 
marked), pericardial effusion (none, non-compressive, 
compressive), respiratory variation in inferior vena cava 
diameter (none, minimal, large), and visual estimation of 
the left ventricular ejection fraction (normal, decreased, 
markedly decreased) (Table 1). We considered that right 
ventricular dilation was marked when its area was 
greater than or equal to the area of the left ventricle 
[41]. We considered that respiratory variation in inferior 
vena cava diameter was large when there was a marked 



visual inspiratory increase in inferior vena cava diameter 
assessed in M-mode. Each semi-quantitative measure- 
ment was rated as follows: 0, disagreement with the 
experts regarding presence of the abnormality; 1, agree- 
ment with the experts regarding presence but disagree- 
ment regarding severity of the abnormality; and 2, full 
agreement with the experts. The maximum total score 
was 20 (Table 1). The hemodynamic diagnosis and sug- 
gested treatment together was rated 1 when fully con- 
sistent with the experts' conclusions and as 0 otherwise. 
TTE scoring and rating of the diagnosis and suggested 
treatment were performed offline by both experts work- 
ing together. The time from the beginning of TTE and 
suggestion of a treatment was recorded. 

Statistical analysis 

Normality of data distribution was tested using the 
Kolmogorov-Smirnov normality test. Variables were sum- 
marized as percentages, means and standard deviations, 
or medians (25% to 75% interquartile range (IQR)), as 
appropriate. 

The total score was compared between novices and 
skilled residents at Ml using the Mann- Whitney L/-test. 
Within the novice group, we compared data at Ml, M3, 
and M6 using Friedman's test for quantitative variables 
and the chi-square McNemar's test for qualitative vari- 
ables. The correlation between scores and number of 
supervised TTEs was assessed using a logarithmic re- 
gression model. 
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Residents were separated into two groups based on 
whether their diagnosis/treatment rating was 1 or 0 
(agreement versus disagreement with the experts), and the 
scores were compared between these groups using the 
Mann-Whitney U-test. In the overall population of nov- 
ices and skilled residents, we plotted a receiver-operating 
characteristics (ROC) curve to identify the score cutoff 
that predicted full agreement between residents and ex- 
perts regarding the diagnosis and treatment (rating of 1). 
The best cutoff was defined as the value providing the 
highest Youden index. Sensitivity and specificity were 
computed as means (95% confidence interval (CI)). 

P values < 0.05 were considered significant. Statistical 
tests were performed using Medcalc 11.6.0 software 
(MedCalc, Mariakerke, Belgium). 

Results 

Patient characteristics 

Each of the 16 residents was evaluated at each of the three 
time points (Ml, M3, and M6), for a total of 48 TTEs, in 
48 different patients. Table 2 lists the main patient charac- 
teristics and TTE findings. Among the 48 patients, no 
patients had acute respiratory distress syndrome, 45% 
were receiving catecholamines and 10% neuromuscular 
blockers at the time of the TTE evaluation. 

Sensitivity of the scoring system to change 

The median total number of supervised TTEs performed 
by each novice was 7 (5 to 10) at Ml, 27 (24 to 30) at M3 
and 67 (57 to 80) at M6 (Table 3). At Ml, the median total 
score was significantly higher in the group of skilled resi- 
dents than in the group of novices (18 (16 to 19) versus 13 
(10 to 15), respectively, P = 0.03) (Figure 1). 

Table 2 Characteristics of the 48 transthoracic 
echocardiography evaluations 



Patient characteristics 



Septic shock 


26 (54%) 


Cardiogenic shock 


7 (15%) 


Hypovolemic shock 


15 (31%) 


Norepinephrine 


16 (33%) 


Epinephrine 


4 (8%) 


Dobutamine 


2 (4%) 


Neuromuscular blockers 


5 (10%) 


Main TTE findings 


Decreased left ventricular ejection fraction 


15 (31%) 


Respiratory variation in IVC diameter 


16 (33%) 


Non-compressive pericardial effusion 


16 (33%) 


Right ventricular dilation 


8 (17%) 


N=48. 




Data are expressed as n (%). 




IVC: inferior vena cava; TTE: transthoracic echocardiography. 





In the novices, the total score increased significantly 
from Ml to M6 (Table 3, Figure 2). The image-quality part 
of the score increased significantly from Ml to M6, 
whereas the semi-quantitative measurement part showed 
a trend toward an increase from Ml to M6 (Table 3). At 
M6, 83% of novices obtained a total score > 15/20 points. 

In the novice group, the percentage of optimal TTE 
views was significantly higher at M6 than at Ml for the 
inferior vena cava view (75% versus 17%, respectively; 
P = 0.02) but not for the apical and parasternal views. 
The number of supervised TTEs correlated significantly 
with the total TTE score (r = 0.68, P< 0.0001) (Figure 3). 
The proportion of novices who needed less than 15 mi- 
nutes to perform TTE and who gave the right diagnosis 
and treatment increased significantly from Ml to M6 
(0% versus 50%, respectively, P = 0.03) (Table 3). 

Ability of the scoring system to predict a correct 
diagnosis and treatment 

In the group of novices, agreement with the experts re- 
garding the diagnosis/treatment improved significantly 
from Ml to M6 (Table 3). At M6, 10 (83%) of the 12 
novices gave the right diagnosis and treatment. 

In the overall population of 16 residents, the total 
TTE score was significantly higher in the subgroup that 
gave the right diagnosis and treatment compared to the 
other subgroup (17 (16 to 18) versus 13 (12 to 16), re- 
spectively, P = 0.002) (Figure 4). 

The ROC curve plotted using all 48 TTE evaluations 
in the 16 residents showed that a total score of 14/20 
was the best cutoff for predicting full agreement regard- 
ing the diagnosis and treatment between residents and 
experts (area under the ROC curve, 0.78; P < 0.0001 ver- 
sus 0.50, sensitivity 81% (95% CI, 64 to 93%) and specifi- 
city 69% (95% CI, 41 to 89%)) (Additional file 1). A total 
score > 19/20 points had 100% specificity (95% CI, 79 to 
100%) for full agreement with the experts regarding the 
diagnosis and treatment. 

Discussion 

In this study, we developed and validated a scoring sys- 
tem for assessing basic critical-care TTE skills in venti- 
lated critically ill patients. A cutoff of 19/20 points had 
100% specificity for agreement between residents and 
experts regarding the diagnosis and treatment. 

In recent years, echocardiography has gained wide- 
spread acceptance among intensivists for the manage- 
ment of acute circulatory failure [29]. Echocardiography 
allows a rapid evaluation of the hemodynamic status and 
is recommended as a diagnostic tool in all patients with 
persistent circulatory failure despite adequate fluid ther- 
apy [26]. There is general agreement that echocardiog- 
raphy to assess patients with circulatory failure can be 
performed reliably by operators who have a lower skill 
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Table 3 Scores for basic critical-care transthoracic echocardiography obtained by the novices at Ml, M3, and M6 





Ml 


M3 


M6 


P-value 


Number of TTEs performed since start of training 


7 (5 to 10) 


27 (24 to 30) 


67 (57 to 80) 


0.0003 


Total Score (/20) 


13 (10 to 14) 


15 (12 to 16) 


17 (15 to 18) 


< 0.001 


Score on the first part of the scoring system (image quality/12) 


7 (5 to 9) 


8 (7 to 9) 


10 (9 to 11) 


< 0.001 


Score on the first part of the scoring system (semi-quantitative measurements/8) 


6 (5 to 7) 


7 (5 to 8) 


7 (6 to 8) 


0.07 


Time from starting TTE to suggesting the diagnosis/treatment 










< 15 minutes 


0 (0%) 


0 (0%) 


6 (50%) 


0.03 


15 to 20 minutes 


0 (0%) 


4 (33%) 


4 (33%) 


0.1 


< 20 minutes 


12 (100%) 


8 (67%) 


2 (1 7%) 


0.002 


Agreement with experts concerning diagnosis/treatment 


0.6 ± 0.5 


0.5 ± 0.5 


0.8 ± 0.4 


< 0.0001 



N = 12. 

Quantitative data are expressed as mean ± standard deviation or median (25% to 75% IQR) range and qualitative data as n I 
M: months; TTE: transthoracic echocardiography. 



level than that required in cardiology [29]. This basic 
skill level can be taught easily, as demonstrated in many 
studies [9-11,20,30-35,42-45]. However, tools are needed 
to evaluate the skill level achieved by trainees. Experts 
agree that the skill level does not correlate reliably with 
the number of supervised echocardiograms [29], as per- 
formance could vary with ease of learning, quality of 
teaching, and specialty of the trainees. Two studies by 
the same group validated a scoring system for assessing 
advanced critical-care TEE [37,38]. 

We developed a score derived from this previously re- 
ported TEE scoring system [37,38]. Our score was de- 
signed for TTE and assessed basic critical-care skills, 
which are the ability to measure semi-quantitative pa- 
rameters but not quantitative parameters [46]. Further- 
more, in contrast to the advanced TEE score, our basic 
TTE score did not include the diagnosis and suggested 



Total score at M1 
20- 
18- 
16- 
14- 
12- 
10- 



P=0.03 



Novices 



Skilled residents 



Figure 1 Total transthoracic echocardiography scores at Ml in 
the 12 novices and 4 skilled residents. The box shows the 25th 
and 75th percentiles, the line in the box the median, and the 
whiskers the 5th and 95th percentiles. 



treatment. We were therefore able to correlate score 
values to agreement between trainees and experts re- 
garding the diagnosis and treatment, as a means of 
validating the scoring system. We found that a score 
cutoff of 19/20 had 100% specificity for full agreement 
between residents and experts regarding the diagnosis 
and treatment. 

We validated our scoring system in three ways. First, 
at Ml, the total score differed significantly between 
residents with and without prior echocardiography ex- 
perience. Second, the total score obtained by novices im- 
proved significantly over the six-month teaching period. 
Third, the total score obtained by the overall population 
of residents was significantly associated with the ability 
to make the correct diagnosis and to suggest the appro- 
priate treatment, as defined using the opinion of two ex- 
perts as the reference standard. 



Total score 
20 . 
18 . 
16 . 
14 . 
12 - 
10 - 



P<0.001 

I 



M1 



M3 



M6 



Figure 2 Total transthoracic echocardiography scores at Ml, 

M3, and M6 in the 12 novices. The box shows the 25th and 75th 
percentiles, the line in the box the median, and the whiskers the 5th 
and 95th percentiles. 
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Number of TTEs 



10 20 30 40 50 60 70 80 90 100 



Figure 3 Correlation between the total transthoracic echocardiography 
(TTE) score and the number of supervised TTEs performed by the 
novices at Ml, M3, and M6. N = 33. The line is the correlation line; 
total scores at M1 are shown as squares, at M3 as circles, and at M6 
as triangles. 



Our findings are consistent with previous evidence 
that acquiring basic critical-care echocardiography skills 
requires only a limited amount of training [9-11,30-35]. 
The theoretical teaching in our study lasted only two 
hours. By M6, after performing a total of 67 supervised 
TTEs, 83% of novices obtained scores > 15/20 points 
and 83% indicated the correct diagnosis and treatment. 
The time from starting TTE to suggesting a treatment 
also decreased considerably over the six-month training 
period. Interestingly, although the novices improved 
their ability to determine the right diagnosis and treat- 
ment between Ml and M6, only the quality of TTE 
views increased significantly over this period, whereas 
the improvement in semi-quantitative measurement 
skills fell short of statistical significance. Therefore, the 



Total score 
20 - 
18 - 
16 - 
14 - 
12 - 



10 - 



P=0.002 



Disagreement for 
diagnosis and/or treatment 



Agreement for 
diagnosis and treatment 



Figure 4 Total transthoracic echocardiography score according 
to agreement for both the diagnosis and the treatment between 
residents (N = 16) and experts. The box shows the 25th and 75th 
percentiles, the line in the box the median, and the whiskers the 5th 
and 95th percentiles. 



improved performance regarding diagnosis and treat- 
ment is probably ascribable to the accumulation of clin- 
ical experience over the six-month rotation rather than 
to specific improvements in TTE skills. An interesting 
feature of our study is that supervised TTEs were 
performed throughout the six-month rotation. This 
practical approach is easy to use in ICUs that have 
board-certified echocardiographers on their staff. 

The first limitation of our study was that only TTEs in 
mechanically ventilated patients were considered. This 
feature improved the standardization of the TTE condi- 
tions and increased the time available for teaching and 
evaluation. However, our score needs to be evaluated in 
non-intubated patients. Second, the number of residents 
included in this study was very low. The score cutoffs 
must therefore be viewed with caution and investigated 
further to determine whether they are suitable for certi- 
fication purposes. Third, the theoretical teaching was 
shorter than the theoretical teaching program currently 
recommended [29,46] and the hands- on training period 
was only six months, which is the duration of resident 
rotations in France. A longer theoretical teaching and a 
longer period of training might have improved the evalu- 
ation of our scoring system. In addition, a significant 
improvement in semi-quantitative measurements might 
have been found over a longer period. Fourth, as resi- 
dents could have access to all patients' medical folders, 
they might be influenced before performing their TTE 
evaluation and we could not exclude the possibility that 
the number of TTE examinations needed to obtain a sat- 
isfactory score might be underestimated. Fifth, residents 
performed only a single TTE examination during their 
evaluation, which therefore might not reflect their true 
TTE proficiencies because of a bias due to the individual 
patient's pathology. Nevertheless, all patients with poor 
echogenicity, as assessed by the experts, were not con- 
sidered for TTE evaluation. Sixth, the scoring system did 
not include the exact items described in a statement by 
two critical-care societies as appropriate for assessing 
basic skills [46]. This statement was published after our 
study was designed. Seventh, we did not previously per- 
form a validation study to first assess the scoring system 
and we cannot state with certainty that two experts was 
an adequate number to carry out the TTE evaluations. 
Nevertheless, in our study, two experts evaluated the 
trainees to avoid the bias induced by only one expert 
analysis. 



Conclusions 

The scoring system that we developed for assessing basic 
critical-care TTE skills is valid and predicts whether 
trainees can identify the correct diagnosis and treatment 
in patients with acute circulatory failure. Further studies 
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are needed to determine whether this score is suitable 
for critical-care echocardiography certification. 

Additional file 



Additional file 1: Receiver-operating characteristics curve generated 
to identify the score cutoff that predicted full agreement between all 
residents (n = 16) and experts regarding the diagnosis and treatment. 
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